Agreement, repeatability, and reproducibility of quantitative retinal layer assessment using swept-source and spectral-domain optical coherence tomography in eyes with retinal diseases

Purpose To evaluate the agreement and precision of retinal thickness measurements obtained using swept-source optical coherence tomography (SS-OCT) and spectral-domain OCT (SD-OCT) in healthy eyes and eyes with retinopathy. Methods This cross-sectional prospective study involved three DRI-OCT Triton (SS-OCT) and three 3D-OCT-1 Maestro (SD-OCT) devices. One of each device (Maestro and Triton) was paired with a single operator. Healthy subjects and patients with retinal diseases were recruited, with study eye and testing order randomized. At least 3 scans per eye were captured for wide scan (12 mm × 9 mm-Triton and Maestro) and macular cube scan (7 mm × 7 mm-Triton, 6 mm × 6 mm-Maestro). Thickness of the full retina, ganglion cell layer + inner plexiform layer (GCL+), and ganglion cell complex (GCL++) were obtained from wide scan and cube scans. Agreement of the measurements between the Triton and Maestro was evaluated by Bland–Altman analysis and Deming regression for each group. Repeatability and reproducibility were assessed using a two-way random effect analysis of variance (ANOVA) model for each parameter by group. Results Twenty-five healthy subjects (25 eyes) and 26 patients with retinal diseases (26 eyes), including, but not limited to, age-related macular degeneration, macular hole, and diabetic retinopathy were recruited. Overall, the measurement differences between Triton and Maestro were <6 μm (mean differences of full retina, GCL++, and GCL+ thickness were ≤5.5 μm, 1.3 μm, and 2.8 μm, respectively) and not statistically significant across the parameters. The repeatability and reproducibility estimates indicate high precision in both devices and groups. Across all the parameters, the repeatability limit was ≤7.6 μm for Triton and ≤12.7 μm for Maestro; reproducibility limit was ≤9.2 μm for Triton and ≤14.4 μm for Maestro. In eyes with retinal pathology, the repeatability coefficient of variation (CV)% was ≤2.6% for Triton and ≤3.4% for Maestro; reproducibility CV% was ≤3.3% for Triton and ≤3.5% for Maestro. Conclusion Both Triton SS-OCT and Maestro SD-OCT provide reliable measurements of retinal thickness in healthy eyes and eyes with retinal diseases. Excellent agreement between the two devices indicates interoperability when testing healthy eyes or eyes with retinal pathology. These findings support the use of thickness measurements from Triton SS-OCT and Maestro SD-OCT in clinical practice.


Introduction
Optical coherence tomography (OCT) is an indispensable imaging technology that enables the capture of in vivo, non-invasive images of the retina and choroid (1).OCT technology has greatly improved over the years since it was first introduced.The development of Fourier domain OCT has facilitated improving both resolution and speed over time-domain OCT (2, 3).The two types of Fourier domain detection are spectral-domain OCT (SD-OCT) and swept-source domain OCT (SS-OCT) (3).SS-OCT devices generally offer higher scanning speed than SD-OCT (4,5) and utilize longer wavelengths, allowing better visualization of deep structures such as the choroid, lamina cribrosa, and sclera (6,7).The high-speed wavelength tuning laser known as swept source, digital data acquisition and processing technology enable SS-OCT to mitigate the signal roll-off observed for SD-OCT (7).The longer wavelength light source also provides greater penetration that renders SS-OCT optimal for imaging eyes with media opacity (7).On the other hand, because axial resolution scales with the square of the central wavelength and the inverse of the bandwidth of the applied light source, SD-OCT devices typically have higher axial resolution than SS-OCT (6).As the two main commercially available OCT systems, each technology has distinct benefits.To facilitate their clinical application and optimization, it is important to thoroughly compare measurements obtained with SS-OCT and SD-OCT systems.
Volumetric scan images and automated retinal thickness measurements generated by the OCT systems are used clinically to qualitatively and quantitatively evaluate the different layers of the retina, facilitating the diagnosis and monitoring of various ocular diseases such as age-related macular degeneration (AMD) and macular edema from various causes (1,8,9).It is now known that many retinal disorders that initially affect the outer retina can lead to subsequent alterations in the inner retina including the retinal ganglion cell (RGC) layer; moreover, degenerative loss of inner retinal layers can occur in early diabetic retinopathy (10).These have resulted in increasing interest in the quantitative assessment of the inner retinal layers, separately from the overall retinal thickness (11).Before one can confidently use measurements from either SS-OCT or SD-OCT in clinical trials or clinical practice, establishing repeatability and reproducibility is critical to judge whether differences between individuals or changes in these measurements over time are of significance (12).Repeatability establishes the variability expected in a measurement over time if nothing changes, while reproducibility considers comparisons of data obtained from different devices and different operators, which might happen in a clinic with multiple instruments, in a multi-site practice, or in the context of multicenter clinical trials (13).A comparison of the repeatability and reproducibility of SS-OCT and SD-OCT as well as an evaluation of the agreement of their measurements would provide essential information regarding their ability to produce reliable examination results and their potential interchangeability in a clinical setting.
The aim of the present study was to evaluate the agreement, repeatability, and reproducibility of thickness measurements of various retinal layers using SS-OCT (DRI OCT Triton, Topcon Inc., Tokyo, Japan) which scans at speeds of 100,000 A scans per second and SD-OCT (3D OCT-1 Maestro, Topcon Inc., Tokyo, Japan) which operates at 50,000 A scans per second, in healthy eyes and eyes with various retinal pathologies.

Methods
The protocol of this prospective study was approved by the IntegReview Institutional Review Boards (3815 S. Capital of Texas Hwy, Suite 320, Austin, TX 78704), and the methodology adheres to the tenets of the Declaration of Helsinki for research involving human subjects and to the Health Insurance Portability and Accountability Act.The recruitment started on March 13, 2017, and ended on April 26, 2017.Written informed consent was obtained from all subjects.

Participants
The following ocular examinations were performed on each eye of each subject to determine eligibility for the study: best-corrected visual acuity (BCVA), refraction, slit lamp biomicroscopy, ophthalmoscopy, intraocular pressure (IOP) and visual field (VF; standard automated perimetry, Humphrey Field Analyzer; 24-2 Swedish interactive threshold algorithm; Carl Zeiss Meditec, Inc., Dublin, California).
In order to be included in this study, subjects had to be 18 years of age or older on the date of informed consent, be able to understand the written informed consent, be willing to participate as evidenced by signing the informed consent and had IOP ≤21 mmHg bilaterally.Exclusion criteria included being unable to tolerate ophthalmic imaging, having ocular media that precluded obtaining acceptable OCT images, having narrow angles that would preclude dilation, and having a history of systemic diseases or medications that might affect the measurements, such as leukemia, dementia or multiple sclerosis, or concomitant use of hydroxychloroquine or chloroquine.
Healthy subjects (Healthy group) had BCVA 20/40 or better and normal ocular health bilaterally (non-visually impairing cataract was acceptable).Both eyes must have met all normal eligibility criteria prior to study eye randomization.An additional exclusion criterion for the Healthy group was evidence of VF defects consistent with glaucomatous optic nerve damage based on at least one of the following two findings: a cluster of 3 or more points in an expected location of the pattern deviation (PD) depressed below the 5% level, at least 1 of which was depressed below the 1% level; or having glaucoma hemi-field test flagged on the VF report as being "outside normal limits." Subjects with retinal disease (Retina group) were included if they were diagnosed with a retinal pathology including, but not limited to AMD, diabetic macular edema, diabetic retinopathy, macular hole, and epiretinal membrane.Subjects were excluded from the Retina group if they had glaucoma, ocular surface disease, or any ocular pathology other than retinal disease in the study eye (non-visually impairing cataract was acceptable).The eye with the specific pathology was deemed the study eye.If both eyes were eligible, then one eye was randomly selected to be the study eye.

Optical coherence tomography scans
In this study, 3 SS-OCT devices (DRI OCT Triton, Topcon Inc., Tokyo, Japan) and 3 SD-OCT devices (3D OCT-1 Maestro, Topcon Inc., Tokyo, Japan) were employed.Three operators were paired with one Triton and one Maestro each to create three distinct operator/ device configurations.Eligible subjects were randomized to select the testing order of the operator/device configuration.All OCT imaging was performed in one session, with breaks at the discretion of the subject.The scan types included wide scan (12 mm × 9 mm-both devices) and macular cube scan (7 mm × 7 mm-Triton, 6 mm × 6 mm-Maestro).For each scan type, a minimum of 3 scans was performed per eye for each configuration of operator/device.Additional scans were acquired by the operator if the scan quality was determined to be unacceptable.Parameter measurements included thickness of the full retina in nine sectors as defined by the early treatment of diabetic retinopathy study (ETDRS), ganglion cell layer (GCL) + inner plexiform layer (IPL) (GCIPL/abbreviated on the instrument reports and in this study to GCL+), and ganglion cell complex (GCC/abbreviated on the instrument reports and in this study to GCL++).
Two imaging experts reviewed each scan independently for image quality acceptance in a randomized fashion.Key aspects of a scan to consider for acceptable image quality were: (1) overall signal strength, (2) local weak signal, (3) poor centration of key structures (fovea not in center of macula scans), (4) eye movements, (5) clipping of the retina (scan is too high or too low and the full retina is chopped off), ( 6) segmentation failure, and (7) improper placement of the macula grid.Acceptable quality scan passed each of these image quality tests.All scans deemed unacceptable were not included in the data analysis.

Statistical analysis
Continuous variables were analyzed using descriptive statistics, including total number (n), mean, standard deviation (SD), and median, and categorical variables were summarized using percentages.
Agreement analysis of OCT B-scan image quality between the Triton and Maestro was performed by group.Cross-tabulation of the grading results between the study devices was provided by grader and group.
The first acceptable scan from each scan type (wide scan and macular cube scan) from the Triton and Maestro was used for agreement analyses.Bland-Altman analysis was used to calculate the mean difference and limits of agreement (LOA), and Deming regression was used to calculate the intercept and slope for linear fitting model.
All acceptable scans from the Triton and Maestro were used in the precision analysis, which was based on a two-way random effect analysis of variance (ANOVA) model.This ANOVA model included the operator/device, eye, and interaction between operator/device and eye as variance components.The repeatability and reproducibility limits and coefficient of variation in percentage (CV%) were calculated as follows: repeatability SD = square root of the residual variance; reproducibility SD = square root of the sum of the operator/device variance, the interaction variance, and the residual variance; repeatability limit = 2.8 x repeatability SD; reproducibility limit = 2.8 × reproducibility SD; repeatability CV% = (repeatability SD)/intercept × 100%; reproducibility CV% = (reproducibility SD)/intercept × 100%.
The sample size was determined based on the 95% LOA and the two-way random effect ANOVA model for precision.To acquire at least 90% power at a one-sided significance level of 5% using an F-test to detect a variance of operator/device effect that was 50% of the total variance, it was determined that a sample size of 21 eyes per population was sufficient.Statistical software SAS 9.3 (SAS Institute, Cary, North Carolina) was used for all calculations.p-values less than 0.05 were considered statistically significant.

Results
Eleven subjects did not meet the eligibility criteria and one subject withdrew consent.Twenty-five healthy subjects (25 eyes) and 26 retinal disease subjects (26 eyes) were included.Demographic and ocular characteristics of the study subjects are summarized in Table 1.Overall, good agreement of B scan image quality between the Triton SS-OCT and Maestro SD-OCT was found.
In general, Bland-Altman analysis showed that the measurement differences between Triton and Maestro were less than 6 μm across the parameters.Deming regression showed most of the slopes of the fitting model between Triton and Maestro were close to +1, and most of the 95% confidence intervals (CI) for intercept contained 0 and CI for slope contained 1.
Specifically, for the thickness of different retinal layers, the mean measurements were similar from the corresponding scan types of the two devices [Triton wide scan vs. Maestro wide scan (12 mm × 9 mm for both) and Triton macular cube scan (7 mm × 7 mm) vs. Maestro macular cube scan (6 mm × 6 mm)].Triton showed slightly lower measurements of the full retina (Table 2, absolute value of mean   S1, absolute value of mean differences ≤2.8 μm) thickness compared to the Maestro in both healthy eyes and eyes with retinal disease.For GCL++ thickness (Table 3), Triton showed higher measurements from the wide scan and lower measurements from the macular cube scan in eyes with retinal disease.These measurement differences were small (mean difference <6 μm) and not statistically significant.Overall, excellent agreement was found across all measurements in both groups.Representative Bland-Altman plots and Deming regression plots are shown in Figures 1, 2 indicating good agreement of full retinal thickness measurements from both wide scan and macular cube scan between Triton and Maestro in eyes with retinal disease.Overall, the repeatability and reproducibility estimates (reproducibility/repeatability limit and CV%) indicate high precision in both devices.Table 4 presents the repeatability and reproducibility for the full retinal thickness of the Healthy and Retina groups.The CV%s range between 0% and 1% for the Triton device; in comparison, Maestro had higher CV%s with a maximum of 1.8%.When comparing to the Retina group, the precision estimates were generally lower in the Healthy group with repeatability limits less than 4 μm and reproducibility limits less than 6 μm for Triton, and repeatability limits less than 10 μm and reproducibility less than 12 μm for Maestro.In general, the wide scan measurements had slightly inferior repeatability/reproducibility limits and CV%s compared with the macular cube measurements.Supplementary Table S2 and Table 5 summarize the repeatability and reproducibility estimates of GCIPL/GCL+ and GCC/GCL++ thickness measurements in Healthy and Retina groups.The repeatability and reproducibility of GCIPL/GCL+ thickness measurements were both good in Triton and Maestro (Supplementary Table S2).For both devices, all the limits were less than 7.2 μm.As expected, the repeatability and reproducibility limits of the Retina group were generally higher than that of the Healthy group.In the Healthy group, all the repeatability and reproducibility CV%s of both Triton and Maestro ranged between 0% and 1% with only 1 exception.In the Retina group, the range of repeatability CV%s of wide scan was 0.8% to 2.6% for the Triton and 1.1% to 3.3% for the Maestro; and the range of repeatability CV%s of macular cube scan was 0.7% to 1.9% for the Triton and also 0.7% to 1.9% for the Maestro.The same trends were found for the reproducibility CV%s.
Similarly, the repeatability and reproducibility of GCC/GCL++ thickness measurements were both good and mostly comparable between the two devices (Table 5).All the limits of repeatability and reproducibility were less than 10 μm, and the limits of the Retina group were higher.In the Healthy group, all the repeatability and reproducibility CV%s of both  Triton and Maestro were ≤1%; CV%s of the Retina group were higher-the maximum repeatability and reproducibility CV%s for the Triton were 1.1% and 1.5% for the wide scan, and 1.1% and 1.3% for the macular cube scan, respectively; and for the Maestro, the maximum repeatability and reproducibility CV%s were 3.0% and 3.0% for the wide scan, and 1.8% and 2.1% for the macular cube scan.

Discussion
This study found excellent agreement and good repeatability and reproducibility of the full retina, GCIPL/GCL+, and GCC/GCL++ thickness measurements obtained using the Triton SS-OCT and Maestro SD-OCT in healthy eyes and eyes with retinal disease.
Establishing the margins of repeatability and reproducibility for OCT-based measurements is important to support use of such metrics   Clinicians can then define a threshold to recognize when a true change in a condition has occurred, which may then be employed in clinical practice to decide whether further treatment is necessary or to identify a therapeutic response (14,15).Reproducibility is relevant to multicenter clinical trials because measurements acquired at multiple sites with various operators and devices will be assessed, but it is also relevant at clinics that have multiple devices and operators, or practices where a patient may visit satellite locations (13).Because SS-OCT and SD-OCT are the two main commercially available OCT systems in retinal practices, it is crucial to understand if they can provide reliable measurements and to evaluate the underlying interchangeability.Several studies have investigated the agreement of thickness measurements between SS-OCT and SD-OCT.Lee et al. ( 16) evaluated agreement of the measurements from the macular cube scan between SS-OCT (DRI-OCT, Topcon) and SD-OCT (Cirrus HD-OCT, Carl Zeiss Meditec) in healthy eyes, and found that SD-OCT yielded larger GCIPL thickness values than SS-OCT, which is consistent with the current study finding that Maestro showed higher thickness measurements.This is expected because a smaller measurement region (Triton 7 mm × 7 mm vs. Maestro 6 mm × 6 mm) contains a relatively larger proportion of thicker retinal areas (16).However, contrary to this study, the study by Lee et al. (16) showed that the differences in thickness measurements between the DRI-OCT and Cirrus HD-OCT were statistically significant, with a mean difference that was approximately 5 fold larger than the current study.This discrepancy could be mainly due to the obvious different measurement regions in their study, which includes a 6 mm diameter circle for the DRI-OCT and a 4.8 mm × 4 mm elliptical annulus for Cirrus.This assumption has been confirmed by a study by Yang et al. (17).Yang et al. (17) found excellent agreement of GCIPL thickness between SS-OCT wide scan (DRI-OCT, Topcon) and SD-OCT macular cube scan (Cirrus HD-OCT, Carl Zeiss Meditec) for healthy eyes using identical measurement regions and segmentations between the two devices (17).To sum these studies up, agreement of OCT measurements mainly depends on the segmentation algorithm (18) and the measurement region/grid.Retinal pathology adversely affects the agreement, consistent with findings from this study, in which eyes with retinal disease had larger measurement differences between the Triton and Maestro.
Despite the good repeatability and reproducibility of both devices in our study, we found several factors that appeared to affect retinal thickness measurement precision.First, as expected and consistent with other studies (13), the repeatability and reproducibility in eyes

FIGURE 1
FIGURE 1 Agreement of full retina thickness measurements from the wide scan between Triton SS-OCT and Maestro SD-OCT in eyes with retinal disease.(A) Bland-Altman plots show all the average measurement differences between the two devices are less than 4 μm.(B) Deming regression plots of full retina thickness from the wide scan.The plots illustrate the fitted linear models (red line) and the identity lines (Triton measurement = Maestro measurement, slope = 1) (black line).Intercepts and slops are shown as mean (95% confidence interval).All slopes are very close to +1.

FIGURE 2
FIGURE 2 Agreement of full retina thickness measurements from the macular cube scan of Triton SS-OCT and Maestro SD-OCT in eyes with retinal disease.(A) Bland-Altman plots show all the average measurement differences between the two devices are less than 6 μm.(B) Deming regression plots of full retina thickness from the macular cube scan.The plots illustrate the fitted linear models (red line) and the identity lines (Triton measurement = Maestro measurement, slope = 1) (black line).Intercepts and slopes are shown as mean (95% confidence interval) indicate good agreement of the measurements between Triton and Maestro.

TABLE 1
Demographics and ocular characteristics of study subjects.The Retina group was older in age and had worse BCVA than the Healthy group.Retinal disease diagnoses included macular degeneration (n = 10), epiretinal membrane (n = 12), macular hole (n = 3), cystoid macular edema (n = 3), diabetic retinopathy (n = 1), and other retinal diseases (n = 15).The Retina group study eyes may have had more than one diagnosis.

TABLE 2
Full retinal thickness agreement between Triton and Maestro.

TABLE 3
Ganglion cell complex thickness agreement between Triton and Maestro.

TABLE 4
Repeatability and reproducibility of full retina thickness measurements.

TABLE 5
Repeatability and reproducibility of ganglion cell complex thickness measurements.